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Executive Summary 



This report presents results from the fifth annual survey of Advanced Technological 
Education (ATE) projects, centers, and articulation partnerships. ATE has 
approximately 220 active awards. Of these, 163 ATE-funded projects, centers, and 
articulation partnerships were asked to participate in the 2004 survey. During the 
survey administration period, 5 projects were removed from the sample, resulting in a 
final target sample of 1 58 grantees. Of these, 1 54 (97%) responded to all or portions of 
the survey. 

Data from this and previous years’ surveys show that the ATE program is being 
implemented as intended by its designers and that it is making a substantial impact in 
its target areas — materials development, professional development, and program 
improvement. Volumes II and III of this report detail the roles that ATE centers and 
projects, respectively, have played during the past 12 months in terms of their activities 
and productivity. 

Where Volumes II and III focus on program implementation and accomplishments, this 
volume examines the program’s primary design elements. Here the intention is to help 
program officers and others determine what design aspects of the program work most 
effectively and identify ways in which to continue to strengthen the program. This 
volume does not focus on determination of merit or worth of the program itself. 

This volume of the 2004 survey report examines the ATE program from the perspective 
of 4 significant program design elements. These elements are described in the ATE 
Program Solicitation (NSF-02-035) and are enumerated below. 

1 . ATE primarily impacts associate degree-level institutions, but also is designed to 
impact secondary schools and 4-year colleges. 

2. ATE funds targeted projects, but also funds comprehensive centers 

3. ATE projects and centers collaborate with other institutions to achieve their 
objectives. 

4. ATE funds articulation partnerships that specifically create pathways to higher 
education for secondary and two-year college students. 

The following key evaluation questions guided this report; 

Does the ATE program design, as enumerated above, have merit in terms of 
promoting program goals? If so, what evidence supports the design? If not, what 
are plausible design alternatives given the available evidence? 




For this report, we limited the evidence for answering the key evaluation questions to 
analysis of the data collected from the 2004 annual survey. These data represent a 
point-in-time accounting of productivity by ATE grantees in each of the primary program 
activities. These data include information about program outcomes through measures 
of impact \n three areas: 

• materials development (number of materials disseminated and cost per material 
item disseminated) 

• professional development (number of persons reached for professional 
development and cost per person reached) 

• program improvement (number of institutions reached and number of students 
reached) 

These measures were derived from items on the 2004 annual survey. As a result, none 
of our measures addressed quality, effectiveness, or sustainability of the efforts of ATE 
projects, centers, and articulation partnerships. Also, it is important to understand that 
measures of productivity reflect activity for the 12 months immediately prior to the 
survey administration period, i.e., roughly March 2003 through April 2004. 

For each of the four program aspects we raised questions and looked at correlation 
findings that suggest answers. Because our findings are based on relationships (e.g., 
correlations), they should be used with care. Flowever, we believe that these results 
provide valuable descriptive information about the program’s design and suggestions for 
actions that can impact overall programmatic directions and results. 

Location of Attention: Associate, Secondary, and Baccaiaureate impact 

Question: Which education levels receive ATE program support? 

Answer: The associate degree level gets most support. Impact occurs at all three 
levels: most at the associate degree level followed by secondary schools and 
baccalaureate institutions. Baccalaureate institutions are a distant third in both program 
emphases and impact. 

Associate degree institutions by far receive the greatest attention in each of the primary 
program areas — materials development, professional development, and program 
improvement — with upwards of 80 percent of grantees working at the associate degree 
level. This is consistent with the Congressional mandate to the program. 

A second perspective on impact across education levels is seen in the proportion of 
program impact in each area. Our findings indicated that 80 percent of materials 
development efforts and three-fourths of the students impacted are at the associate 
degree level. Flowever, less than half of professional development participants and 
institutions impacted were at the associate degree level. In these latter categories. 




impact at the secondary level was comparable to that at the associate degree level. In 
all categories, impact at the baccalaureate level was substantially less than at the other 
levels. 

Question: Is the impact made by projects and centers in each program area related to 
the number of different education levels where they work? 

Answer: It depends. Grantees engaged in materials development make a 
proportionally greater impact if they focus on one education level. Grantees engaged in 
professional development make a proportionally greater impact if they focus on all three 
levels; and grantees focused at two adjacent education levels are proportionally the 
most successful in directly impacting students. 

Our findings suggest that depending on a grantee’s area of focus — e.g., materials 
development versus program improvement — working at different education levels can 
be advantageous or disadvantageous. For example, materials development impact, in 
terms of distributing copies of materials, is proportionally the greatest for those grantees 
that work to develop materials at one education level. Conversely, professional 
development impact, in terms of number of participants, is proportionally greatest for 
those projects that work across all three education levels. Grantees that work at two 
adjacent education levels tend to demonstrate a proportionally smaller impact in 
materials distributed and professional development participants and a proportionally 
greater impact in the number of students impacted. 

Type of Award (Projects vs. Centers) 

Question: What is the relative impact of ATE projects and centers with respect to 
productivity in each of the ATE program areas? 

Answer: Projects produce greater total impacts, but proportionally, centers are more 
productive when aggregate funding for projects and centers is considered. 

Centers received a bit more than a third of total funding provided to ATE grantees — 
$36.3 million for centers versus $68.9 million for projects. They produced about one- 
fourth of the impact for the outcome related to dissemination of materials, but produced 
nearly half of the impact in professional development and program improvement 
measures. Center results are much stronger than those of projects in terms of total 
student impact — 57 percent versus 43 percent. 

Question: Are centers or projects more cost-effective in directly impacting students? 

Answer: Centers appear to be more cost-effective than projects in impacting students. 

We focused on direct student impact (i.e., number of students enrolled in at least one 
ATE-supported course) in evaluating the cost-effectiveness of projects and centers 
because this is the best available measure to assess whether or not ATE is making a 
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direct impact on the workforce. Our analysis showed that when funding for the most 
recent 12 month period is considered, centers impact more students per NSF dollar 
spent — centers are awarded roughly $400 for each student impacted, while projects 
receive in excess of $1 ,000 for each student impacted. 

Question: Is the impact made by projects in each program area related to the number 
of different program activities in which they engage? 

Answer: It depends. Projects engaged in materials development make a greater 
impact if their activities are more focused. However, professional development and 
program improvement impact does not appear to be related to the number of activities 
in which a project is engaged. 

Given that projects, overall, appeared to make a proportionally lower impact and were 
less cost-effective than centers, we hypothesized that this may be due to projects 
attempting too many different activities given the limited ATE funding they receive. We 
referred to this issue as specificity of purpose. However, our findings suggest that the 
benefit derived from specificity of purpose is not pervasive across program areas and 
that it may be related to the emphasis of an ATE project. For example, projects that 
focus on materials development appear to be more productive in distributing their 
materials if they are focused on one or two program activities. Conversely, projects that 
focus on professional development or program improvement appear to be more 
productive if they focus on three or four different program activities. 

Number and Types of Collaborations 

Question: What is the relative impact of collaboration (number of collaborators and 
monetary and in-kind support received) on project outcomes? 

Answer: The number of collaborators is positively related to impact, as defined in our 
analysis. 

We found that the total number of collaborators predicts from roughly 10 percent to a 
third of the variance for several impact outcomes (Pearson r correlation coefficients are 
reported; r^ equals proportion of variance accounted for): 

Regarding Materials 

• total number of copies of materials distributed by a project (r = .31 , p = .01 8) 

• number of institutions using at least one material (r = .58, p = .000), 

• the self-perception that a project is successful in achieving the ATE program goal 
of national distribution of developed materials (r = .32, p = .003) 

Regarding Professional Development 

• the total number of professional development participants (r = .50, p = .000) 
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Regarding Program Improvement 



• the total number of students impacted (r = .31 , p < .01 ) 

• the extent to which the project perceives its work as a model for program 
improvement (r = .23, p < .05) 

• the extent to which the project perceives that its program is being broadly 
disseminated (r=.25, p < .05). 

Neither monetary nor in-kind support helps explain variability among numbers of (a) 
materials disseminated, (b) persons reached for professional development, or (c) 
number of students reached for instruction. 

Question: Is impact related to the types of organizations with which the project 
collaborates? 

Answer: Yes. 

Each of the three productivity measures is significantly related to the types of 
collaborating organizations (the fourth measure, institutions, was not related to numbers 
and types of collaborators). Materials distribution, professional development 
participants, and students impacted are predicted with R-squares of .69, .77, and .32, 
respectively. Standardized regression coefficients from this analysis indicate that the 
nature and strength of these relationships vary markedly across the three dependent 
variables. Those findings indicate that 

• Projects with more collaborators among other ATE awards disseminate more 
materials. 

• Projects with large numbers of collaborators within their host institution and among 
other education institutions had larger numbers of participants in their ATE- 
sponsored professional development activities. 

• Projects that directly impacted large numbers of students had larger numbers of 
collaborations with other education institutions. 

Perhaps the most striking aspect of these findings is that collaboration with 
business/industry does not help to predict any of the three dependent variables. The 
caveat on this is that none of the impact measures used in this analysis addressed 
quality, effectiveness, or sustainability. Some of our other findings suggest that 
collaborations with business and industry primarily serve as means to improve the 
quality of materials and content for instructional programs. That is, business and 
industry collaborations may serve other outcomes than those reported here. 

Question: How can the amount of collaboration be increased? 
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Answer: Both award size and advisory panel investment are modest predictors of the 
level of collaboration. 

Because the nature and amount of collaborations are positively related to higher levels 
of program impact, as defined in this study, we examined several organizational factors 
in relation to levels of collaboration — i.e., number of collaborators and the amount of 
monetary and in-kind support. They were (1 ) the total amount of grant funding, (2) the 
maturity of a particular grant, (3) investment in advisory panel activities, and (4) 
investment in evaluation. We hypothesized in this analysis that higher levels of each 
organizational practice indicator would be related to higher levels of collaboration. 

We found that award amount is positively related to the number of collaborators (r=.45, 
p<.05), amount of supplemental monetary support received (r=.26, p<.05), and amount 
of in-kind support received (r=.30, p<.01); this is consistent with our hypothesis. 
Additionally, the investment in advisory panels was found to be modestly related to the 
number of collaborators (r=.24, p<.05); and both investments in advisory panels (r=.30, 
p<.05) and evaluations (r=.30, p<.05) were modestly related to amount of monetary 
support. 

This group of factors, while small in amount of variance accounted for, does give some 
indication of ways to bump up collaborative efforts. For example, these findings 
suggest that there is merit to the argument that larger projects and centers are better 
able to leverage their ATE grant funds to attract additional support and that investment 
in advisory panels and evaluation may contribute to additional funding opportunities, 
perhaps through the network established by an advisory panel or the evidence of impact 
and effectiveness developed by an evaluator. 

Articulation Partnerships as Pathways for Student Progress 

To date, a relatively small number of articulation partnerships have been funded — 8 of 
1 54 respondents to this year’s survey fell in this category. However, results from this 
year’s survey clearly show that many traditional projects and centers are engaged in 
establishing articulation agreements even though they are not specifically categorized in 
this program track. We used those other projects and centers as comparison points for 
the articulation partnerships. 

Question: What is the relative contribution of articulation partnerships, versus projects 
and centers, in promoting articulation of students between 2-year and 4-year colleges? 

Answer: Their contribution is greater than would be expected from their small number 
of grants. 

Articulation partnerships received 3 percent of the total funding awarded to ATE 
grantees that reported on the creation of articulation agreements (projects received 59 
percent and centers 38 percent). However, articulation partnerships make a 
disproportionately large impact in terms of involving institutions (12 percent of impact) 
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and impacting students (17 percent of impact). This is despite having a relatively small 
number of agreements in place. 



Question: Do articulation partnerships offer any cost benefit advantage over articulation 
agreements established by projects and centers? 

Answer: Preliminary findings suggest that articulation projects are a good value in terms 
of students articulating across education levels. 

We examined the cost-effectiveness of articulation partnerships, projects, and centers in 
terms of (1) establishing agreements and (2) impacting students. This analysis is based 
on our sample where articulation partnerships and projects were, on average, funded 
for 2.9 and 3.1 years, respectively, while centers received funding for an average of 3.3 
years. 

Findings show that articulation partnerships are the most cost-effective mechanism in 
terms of impacting students, receiving $3,600 for every student impacted versus 
$24,600 for projects and $20,000 for centers. However, agreements created by 
partnerships reportedly cost more to create — an average of $120,000 versus $93,000 
for projects and $52,000 for centers. That high cost per agreement is more than offset 
by the much greater productivity in terms of students articulating under these 
agreements. 

These findings support the idea that grantees that receive smaller amounts of funding 
are able to be more productive in a given program area by focusing on fewer activities. 
Of course, this was the rationale for creating a specific program track to promote 
articulation of students across education levels and, in this sense, articulation 
partnerships are unique within the ATE program in that they are expected to be 
narrowly focused. 

Conclusions and Recommendations 

Our analysis, which is summarized above and presented in more detail in the main body 
of this report, enabled us to draw two overarching conclusions about the ATE program 
design. The first conclusion addresses an issue we call “critical mass”; the second 
deals with collaboration. Each is discussed below along with what we believe are 
justifiable recommendations. 



Critical mass. The concept of critical mass, as we choose to operationalize it 
here, refers to the intersection of funding levels and specificity of purpose. In turn, 
specificity of purpose incorporates two elements — program activities and location of 
attention (i.e., work in multiple program areas and work at multiple education levels). In 
theory, these concepts are inversely related in that greater levels of funding can enable 
broader programming at multiple education levels, while lower funding levels require 
greater specificity of purpose, both in programming as well as location of attention. 
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Our analysis supports the assertion that the concept of critical mass is an important 
consideration for the ATE program. Furthermore, critical mass may be more or less 
important for individual grantees, depending on the program activities in which a project 
or center chooses to engage. 

The evidence for this conclusion is summarized below: 

• A relatively small number of centers, which individually receive large amounts of 
funding but overall receive little funding, is proportionally more productive than the 
large number of projects, which individually receive small funding amounts. 

• By a factor of 2.5, centers appear to be more cost-effective than projects in directly 
impacting students ($1 ,000 per student for projects versus $400 per student for 
centers). 

• Articulation partnerships, which were established on the premise of specificity of 
purpose, make a disproportionately high impact on students given the amount of 
funding they receive. 

• Materials development productivity is proportionally greatest for those projects 
engaged in one or two activities and for all grantees if they are focused at only one 
education level. 

• Professional development productivity is proportionally greatest for projects 
engaged in three or four activities and for grantees that work across all three 
education levels. 

• Direct student impact is proportionally greatest for projects that work at two 
education levels and for grantees that work across all three education levels. 

These findings suggest that the ATE program could possibly make a greater impact if it 
were able to provide larger amounts of funding to each grantee and if it encouraged 
grantees to limit their activities and target their efforts in relation to their primary goals 
and objectives (i.e., materials development, professional development, direct student 
impact, or articulation agreements). Of course, we understand that if overall program 
funding levels are held relatively constant, this recommendation could mean a reduction 
in the number of grants awarded under the program — which may not be politically 
feasible. 

Notwithstanding the above, there is substantial evidence to suggest that achieving 
critical mass within an individual grant, either through boosting funding or strategically 
limiting activity and targeting efforts, produces greater impact. 

Collaboration. The ATE program has long stressed the importance and value of 
collaboration toward achieving program goals. The analysis in this report supports that 
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position and provides strong evidence that positive relationships exist between the 
number of collaborators and various measures of productivity, as defined for this report. 
Based on this, we recommend that ATE continue to stress the importance of 
collaboration, provide mechanisms to encourage and facilitate collaboration, and hold 
grantees accountable for establishing and maintaining positive collaborative 
relationships. 

Coupled with the above finding is evidence that suggests collaborations with various 
types of institutions are predictive of productivity in different program areas. Thus, 
depending on a particular grantee’s area of focus, it may benefit from focusing 
collaborative efforts on a particular type of institution. We should stress that these 
findings are preliminary and have been influenced by outlier data points. However, the 
findings also have strong face validity, which suggest that there is merit to the analysis. 
Given this, we would recommend that ATE share these findings with ATE grantees and 
investigate the nature of collaborative activities with different types of institutions to 
determine the degree to which some relationships are more or less beneficial given a 
particular program focus. 

Lastly, there is evidence to support the conclusion that investments in advisory panels 
and evaluation are positively related to levels of collaboration. We recommend that 
ATE establish clear expectations that ATE grantees make dedicated investments in 
both advisory panel activities and evaluation. While the ATE program has long 
recommended and expected that grantees plan and implement these activities, there 
has not been a similar emphasis on what constitutes an appropriate investment in these 
activities. Our recommendation is that ATE grantees assign financial resources to 
these activities that demonstrate their commitment to these activities. For advisory 
panels, this may constitute budgeting for honorariums and all meeting expenses. For 
evaluation, this means, as a rule of thumb, budgeting between 7-10 percent of the grant 
for evaluation purposes. 
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